Simulator, simulation method, and simulation program

ABSTRACT

The system behavior is evaluated by checking the position and the orientation of a target processed by a processing device in accordance with a control instruction. A simulator estimates a behavior of a system including a processing device that processes a target. The simulator includes a measurement unit that performs image measurement of an input image including at least a part of a target as a subject of the image, an execution unit that executes a control operation for generating a control instruction directed to the processing device based on a measurement result obtained by the measurement unit, and a reproduction unit that reproduces, in the system, a behavior of a target detected in the input image together with information about a type and an orientation of the target based on time-series data for the control instruction output from the execution unit and the measurement result from the measurement unit.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims priority to Japanese Patent Application No.2015-225785 filed Nov. 18, 2015, the entire contents of which areincorporated herein by reference.

FIELD

The present invention relates to a simulator, a simulation method, and asimulation program for estimating the behavior of a system.

BACKGROUND

In the field of factory automation (FA), automatic control techniquesusing visual sensors are used widely. Such techniques include automaticprocessing in which an image of a target such as a workpiece iscaptured, the captured image undergoes image measurement such as patternmatching, and various control devices operate based on the measurementresults.

For example, Japanese Unexamined Patent Application Publication No.2012-187651 (Patent Literature 1) describes conveyor tracking involvinga visual sensor and a robot. A system for such conveyor trackingincludes the visual sensor and a robot controller for controlling therobot connected to each other with a network.

Designing or examining the system to be controlled with the aboveautomatic control technique may need preliminary evaluation of theperformance of the entire system. In response to this, a technique hasbeen developed for virtually creating a system and simulating itsoperation. For example, Japanese Unexamined Patent ApplicationPublication No. 2013-191128 (Patent Literature 2) describes a techniquefor integrated simulations of a mechanical system including a visualsensor in a real space corresponding to a virtual imaging unit. With thetechnique described in Patent Literature 2, a 3D simulator and a visualsensor simulator cooperate with each other to virtually generatecaptured images of a workpiece in a 3D space at predetermined timings.

CITATION LIST Patent Literature

Patent Literature 1: Japanese Unexamined Patent Application PublicationNo. 2012-187651

Patent Literature 2: Japanese Unexamined Patent Application PublicationNo. 2013-191128

SUMMARY Technical Problem

For example, the conveyor tracking system described in Patent Literature1 generates a control instruction to each device based on measurementresults transmitted from the visual sensor. When this system issimulated, checking the measurement results from the visual sensor maybe intended, together with checking of the behavior of each device.

When, for example, the conveyor tracking system transfers a workpiece onone conveyor onto another, each workpiece may be oriented in apredetermined direction. In this case, checking whether each workpiecehas been processed correctly may be performed.

Solution to Problem

A simulator according to one aspect of the present invention estimates abehavior of a system including a processing device for processing atarget. The simulator includes a creating unit, a measurement unit, anexecution unit, and a reproduction unit. The creating unit virtuallycreates the system in a three-dimensional virtual space. The measurementunit performs image measurement of an input image including at least apart of a target as a subject of the image in a manner associated with afirst area predefined at a predetermined position in thethree-dimensional virtual space. The image measurement includessearching the input image for a part corresponding to one or more piecesof predetermined reference information. The execution unit executes acontrol operation for generating a control instruction directed to theprocessing device based on a measurement result obtained by themeasurement unit. The reproduction unit reproduces, in the system, abehavior of a target detected in the input image together withinformation about a type and an orientation of the detected target basedon time-series data for the control instruction output from theexecution unit and the measurement result obtained by the measurementunit.

In some embodiments, the type of the target includes informationindicating, among the pieces of predetermined reference information, themost probable degree of correspondence with the target.

In some embodiments, the type of the target includes informationindicating whether a degree of correspondence between the target and thepieces of predetermined reference information satisfies a predeterminedcondition.

In some embodiments, the reproduction unit displays the target using atleast one of a color, a shape, or a size that differ depending on thetype of the target.

In some embodiments, the simulator further includes an input unit thatreceives a setting of at least one of the color, the shape, or the sizethat differ depending on the type of the target.

In some embodiments, the measurement unit outputs, as the measurementresult, a rotation angle of a part corresponding to at least one of thepieces of reference information included in the input image, and thereproduction unit generates information about the orientation of thetarget based on the rotation angle output as the measurement result.

In some embodiments, the reproduction unit displays, in addition to thetarget, an object indicating the orientation of the target.

In some embodiments, the reproduction unit displays a feature added toan appearance of the reproduced target in a manner associated with theorientation of the target.

In some embodiments, the system includes a carrier configured totransport the target, and the predetermined position is on atransporting path of the carrier.

In some embodiments, the reproduction unit sequentially updates adisplay position of the target in the three-dimensional virtual spacebased on information indicating a position or a displacement of thecarrier that transports the target.

A simulation method according to another aspect of the present inventionis implemented by a computer for estimating a behavior of a system. Thesystem includes a processing device for processing a target. The methodincludes a creating process, a measurement process, a simulationexecution process, and a reproduction process. The creating processincludes virtually creating the system in a three-dimensional virtualspace. The measurement process includes performing image measurement ofan input image including at least a part of a target as a subject of theimage in a manner associated with a first area predefined at apredetermined position in the three-dimensional virtual space. The imagemeasurement includes searching the input image for a part correspondingto one or more pieces of predetermined reference information. Thesimulation execution process includes executing a control operation forgenerating a control instruction directed to the processing device basedon a measurement result from the image measurement. The reproductionprocess reproduces, in the system, a behavior of a target detected inthe input image together with information about a type and anorientation of the detected target based on time-series data for thecontrol instruction and the measurement result.

A simulation program according to another aspect of the presentinvention is used to estimate a behavior of a system. The systemincludes a processing device for processing a target. The simulationprogram causes a computer to implement a creating process, a measurementprocess, a simulation execution process, and a reproduction process. Thecreating process includes virtually creating the system in athree-dimensional virtual space. The measurement process includesperforming image measurement of an input image including at least a partof a target as a subject of the image in a manner associated with afirst area predefined at a predetermined position in thethree-dimensional virtual space. The image measurement includessearching the input image for a part corresponding to one or more piecesof predetermined reference information. The simulation execution processincludes executing a control operation for generating a controlinstruction directed to the processing device based on a measurementresult from the image measurement. The reproduction process includesreproducing, in the system, a behavior of a target detected in the inputimage together with information about a type and an orientation of thedetected target based on time-series data for the control instructionand the measurement result.

Advantageous Effects

Embodiments of the present invention allow evaluation of the behavior ofa system by checking both the position and the orientation of a targetprocessed by a processing device in accordance with a controlinstruction, and thus allow the validity of the system to be determinedreadily.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing the configuration of a system tobe simulated by a simulator according to one embodiment.

FIG. 2 is a schematic diagram describing a simulation method implementedby the simulator according to the embodiment.

FIG. 3 is a schematic diagram showing the hardware configuration of thesimulator according to the embodiment.

FIG. 4 is a schematic diagram showing the functional structure of thesimulator according to the embodiment.

FIG. 5 is a schematic diagram showing the functional structure of thesimulator according to a modification.

FIG. 6 is a flowchart showing the procedure of simulation performed bythe simulator according to the embodiment.

FIGS. 7A and 7B are schematic diagrams describing examples of imagemeasurement performed by the simulator according to the embodiment.

FIG. 8 is a schematic diagram describing the measurement results of theimage measurement performed by the simulator according to theembodiment.

FIG. 9 is a diagram showing an example user interface screen forreproducing simulation results provided from the simulator according tothe embodiment.

FIGS. 10A and 10B are schematic diagrams describing an example of usersetting for the display mode of simulation results provided from thesimulator according to the embodiment.

FIG. 11 is a schematic diagram describing another example of usersetting for the display mode of simulation results provided from thesimulator according to the embodiment.

FIG. 12 is a schematic diagram describing another example of usersetting for the display mode of simulation results provided from thesimulator according to the embodiment.

FIGS. 13A and 13B are diagrams each showing an example user interfacescreen for supporting determination of each workpiece type based onsimulation results provided from the simulator according to theembodiment.

FIGS. 14A and 14B are diagrams each showing an example display ofworkpiece orientations based on simulation results provided from thesimulator according to the embodiment.

FIGS. 15A and 15B are diagrams each showing another example display ofworkpiece orientations based on simulation results provided from thesimulator according to the embodiment.

FIG. 16 is a diagram showing another example user interface screen forreproducing simulation results provided from the simulator according tothe embodiment.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described in detailwith reference to the drawings. The same or corresponding components inthe figures are given the same reference numerals, and will not bedescribed redundantly.

A. Overview

A simulator according to the present embodiment estimates the behaviorof a system. More specifically, the simulator according to the presentembodiment can estimate the behavior or other acts of a virtuallycreated system using an input image. Although the simulator simulates asystem including a processing device that processes targets transportedon a transporting path in the embodiment described below, the simulatormay simulate any other system.

In the present embodiment, one or more conveyors function as a carrierfor transporting targets on a transporting path, and one or more robotsfunction as the processing device for processing the targets. Thecarrier and the processing device are not limited to these examples andmay be selected as appropriate depending on the system to be simulated.The targets may hereafter also be referred to as workpieces. A workpiecemay be any item that can be measured by a visual sensor, such as an endproduct or its part, or an intermediate product or its part.

The simulation performed by the simulator according to the presentembodiment will now be described briefly.

FIG. 1 is a schematic diagram showing the configuration of a system tobe simulated by the simulator according to the present embodiment. Asshown in FIG. 1, for example, a conveyor tracking system 1 includes arobot 210, which picks up a workpiece 232 within a predeterminedtracking area 231 when the workpiece 232 transported continuously on aconveyor 230 reaches the tracking area 231, and transfers to and placesthe workpiece 232 onto a conveyor 240. This series of actions performedby the robot 210, or picking, transferring, and placing, may be referredto as the pick-and-place operation. In the pick-and-place operation, therobot 210 may rotate the workpiece 232 picked up from the conveyor 230and orient the workpiece in a predetermined direction.

To enable the pick-and-place operation of the robot 210, an imaging unit222 captures an image of an imaging area 221 defined on a part of theconveyor 230, and a visual sensor 220 performs image measurementincluding pattern matching of the input image captured by the imagingunit 222 and obtains the measurement results including information aboutthe position, type, orientation, and other parameters of the workpiece232.

A controller 200 executes a predetermined control logic based on themeasurement result obtained by the visual sensor 220 to sequentiallyupdate (or track) the position of the workpiece 232 and generate acontrol instruction for the robot 210. The controller 200 typicallyincludes a programmable logic controller (PLC).

To generate the control instruction for the robot 210, the controller200 refers to the status value of the robot 210, and an encoder valueprovided from an encoder 236, which is connected to a drive roller 234for driving the conveyor 230 (encoder value 1), and an encoder valuefrom an encoder 246 coupled to a drive roller 244 for driving theconveyor 240 (encoder value 2). The controller 200 and the visual sensor220 are connected to each other with a network 202 to allow datacommunication between them. The measurement results are transmitted fromthe visual sensor 220 to the controller 200 through the network 202.

Evaluating the processing capability (e.g., a tact time) and theaccuracy of processing may be intended before the conveyor trackingsystem 1 shown in FIG. 1 is installed. This is because actuallyinstalling the conveyor tracking system 1 and checking the processingcapability is often difficult due to the cost or time constraints. Thesimulator according to the present embodiment is designed to achievesimpler estimation of the behavior of the conveyor tracking system 1.

More specifically, the simulator according to the present embodimentvirtually creates a system to be simulated in a three-dimensionalvirtual space, and combines the virtually created system with any inputimage to achieve more efficient simulation.

FIG. 2 is a schematic diagram describing a simulation method implementedby the simulator according to the present embodiment. With reference toFIG. 2, the simulator models the entire conveyor tracking system 2,which is to be simulated, and incorporates an input image into thissystem model to simulate an image-capturing operation performed by theimaging unit 222.

The input image incorporated into the system model of the conveyortracking system 2 can represent specifications expected in the design(e.g., the moving speed of a workpiece 232 or the number of workpieces232 passing per unit time). Typically, the input image may be an imageactually captured on a similar production line.

Although the input image used in the simulation is expected to be animage captured in the existing system (e.g., the system before updatewhen an updated system is to be simulated), the input image may becaptured in any system and situation. More specifically, the input imagemay be any image including chronological change information about atarget to be simulated (typically, a workpiece 232).

The input image may be moving image data or data representing aplurality of still images arranged chronologically. The reproductionspeed of the moving image data or the update frequency of the datarepresenting the still images can be adjusted as appropriate to adjustthe chronological changes (or the moving speed) of a target to becontrolled. Adjusting the input image incorporated in the system modelin this manner allows the simulation to yield an optimal value for thechronological changes of the control target.

Additionally, still images that are not actually captured sequentiallybut are captured in different situations may be arranged aschronologically changing images and used as an input moving image.Although the images generated in this case have no workpieceoverlapping, this causes substantially no problem in performing thesimulation.

Simulating this system including the visual sensor may involveevaluating the workpiece measurement results obtained from the visualsensor and the behavior of the controller based on the measurementresults. More specifically, the pick-and-place operation described abovemay involve evaluation to determine whether the image measurement forthe input image yields correct information about the workpiece (e.g.,the type of the workpiece), or whether the workpiece is oriented in anintended direction through the pick-and-place operation performed by therobot.

A simulator 100 according to the present embodiment can represent, inthe system, information about the type of the workpiece determined inthe image measurement and the orientation and the behavior of theworkpiece. The simulator 100 according to the present embodiment willnow be described in detail.

B. Hardware Configuration of Simulator

The hardware configuration of the simulator 100 according to the presentembodiment will now be described. The simulator 100 according to theembodiment is implemented typically by one or more computers executing aprogram.

FIG. 3 is a schematic diagram showing the hardware configuration of thesimulator 100 according to the present embodiment. With reference toFIG. 3, the simulator 100 is, for example, a computer having thegeneral-purpose computer architecture. The simulator 100 includes aprocessor 102, a main memory 104, an input unit 106, a display unit 108,a network interface 110, a hard disk drive (HDD) 120, an optical drive112, and a communication interface 116. These components are connectedto each other with an internal bus 118 in a communicable manner.

The processor 102 loads a program stored in the hard disk drive 120 intothe main memory 104 and executes the program to implement the functionsand the processing described later. The main memory 104 is a volatilememory and functions as a working memory used for program execution bythe processor 102.

The input unit 106 typically includes a keyboard, a mouse, a touchpanel, and a touchpad, and receives a user operation. The display unit108 includes a display and an indicator, and presents various pieces ofinformation to a user.

The network interface 110 exchanges data with external devices such as aserver through a network. The optical drive 112 reads various programsfrom an optical disc 114 or other media, and installs the programs intothe hard disk drive 120. The communication interface 116 is, forexample, a universal serial bus (USB) communication interface, andexchanges data with external devices such as an auxiliary storagethrough local communications.

The hard disk drive 120 stores an operating system (OS) 122, a programfor providing the functions of the simulator, such as a simulationprogram 124, and an image data group 140 including preliminarilyobtained input images used for simulation.

Although an intended program is installed in the simulator 100 via theoptical drive 112 in the configuration example shown in FIG. 3, aprogram may be downloaded from a server or other devices on the network.

When the simulator is a general-purpose computer as described above, anOS may be installed on the computer to provide the basic function of thecomputer, in addition to a program for providing the functions accordingto the present embodiment. In this case, the simulation programaccording to the present embodiment may call program modules included inthe OS in a predetermined order and/or at predetermined timings asappropriate to perform processing. More specifically, the programaccording to the present embodiment may not include these programmodules and may cooperate with the OS to perform processing. The programaccording to the present embodiment may not include such modules.

The program according to the present embodiment may be incorporated as apart of another program to be combined. The program according to thepresent embodiment may not thus include modules of the program to becombined, and may cooperate with the program to achieve processing. Inother words, the simulation program according to the present embodimentmay be incorporated in the other program.

Although FIG. 3 shows the simulator 100 that is a general-purposecomputer, the simulator 100 may be partly or entirely implemented usinga dedicated circuit (e.g., an application specific integrated circuit,or ASIC). Additionally, an external device may perform a part of theprocessing of the simulator 100.

Although the simulator 100 according to the present embodiment shown inFIG. 3 is a single device, two or more devices cooperating with eachother may function as the simulator 100. The simulator 100 according tothe present embodiment may include a system combining two or moreseparate devices.

C. Functional Structure of Simulator

The functional structure of the simulator 100 according to the presentembodiment will now be described.

FIG. 4 is a schematic diagram showing the functional structure of thesimulator 100 according to the present embodiment. The simulator 100shown in FIG.

4 includes a visual sensor simulator 150, a controller simulator 160, areproduction module 170, a user interface module 180, and an encoderemulator 190 as software functions. This group of functional modules istypically implemented by the processor 102 executing the simulationprogram 124 (refer to FIG. 3).

The user interface module 180 provides an operation screen for aidingthe user to set and generate a setting parameter 152, a control program162, three-dimensional design data 172, and a workpiece display settingparameter 174. The user interface module 180 also provides any userinterface used when the reproduction module 170 displays simulationresults.

The user interface module 180 includes a model creating module 182 forhandling the three-dimensional design data 172. The model creatingmodule 182 virtually creates the system to be simulated in athree-dimensional virtual space. More specifically, the model creatingmodule 182 displays a three-dimensional virtual space, and provides asetting and operation screen for creating the system to be simulated inthe three-dimensional virtual space.

The simulator 100 according to the present embodiment typically createsa system including a carrier (typically, a conveyor) virtually in athree-dimensional virtual space. As shown in FIG. 2, the system modelincludes the visual sensor 220 virtually located at a first position ona transporting path associated with the carrier (conveyor), and thecontroller 200 virtually located at a second position on thetransporting path. The system model further includes an imaging areadefined for the visual sensor. The system model intended for thepick-and-place operation as shown in FIGS. 1 and 2 may further define anarea in which a workpiece is to be picked up (tracking area) and an areain which the workpiece is to be placed.

The visual sensor simulator 150 is a module for simulating theprocessing performed by the visual sensor 220, and performs imagemeasurement of an input image including at least a part of a workpieceas a subject of the image in a manner associated with the imaging areapredefined on the transporting path (conveyor) in the three-dimensionalvirtual space. More specifically, in response to a fetch instruction(typically, a trigger signal) from the controller simulator 160, thevisual sensor simulator 150 retrieves the corresponding image data fromthe preliminarily obtained image data group 140, and performs the imagemeasurement corresponding to the processing performed by the visualsensor 220 (refer to FIGS. 1 and 2). Typically, the image measurementperformed by the visual sensor simulator 150 includes searching theinput image for the part corresponding to one or more pieces ofpredetermined reference information.

The measurement results from the image measurement performed by thevisual sensor simulator 150 are output to the controller simulator 160.The output processing corresponds to the transmission of the measurementresults obtained by the visual sensor 220 to the controller 200 throughthe network 202 in the conveyor tracking system shown in FIGS. 1 and 2.The image measurement in the visual sensor simulator 150 is performed inaccordance with a predetermined setting parameter 152.

The controller simulator 160 performs a control operation for generatinga control instruction for a robot, which is an example of the processingdevice, based on the measurement results from the visual sensorsimulator 150. The controller simulator 160 is a module for simulatingthe processing in the controller 200 (refer to FIGS. 1 and 2), andperforms a control operation (a sequence instruction, a motioninstruction, or various functional instructions) in accordance with thepreliminarily created control program 162. Trace data including inputand output associated with the control operation performed in thecontroller simulator 160 is output to the reproduction module 170 astrace data.

The trace data includes time-series data for the control instructionsoutput from the controller simulator 160 to the robot and themeasurement results from the visual sensor simulator 150.

The control operation performed in the controller simulator 160 includesprocessing for generating a fetch instruction (trigger signal) forretrieving image data, which is to be transmitted to the visual sensorsimulator 150. More specifically, when a predetermined condition issatisfied, the controller simulator 160 generates a trigger signal. Thepredetermined condition is, for example, that the conveyor has moved bya predetermined distance, or a predetermined period has ended. Asdescribed later, the distance by which the conveyor has moved isdetermined based on information generated by the encoder emulator 190.

The reproduction module 170 reproduces, in the system, the behavior ofthe workpiece detected from the input image, together with theinformation about the type and the orientation of the workpiece, basedon the trace data (including time-series data for a control instructionto be transmitted to the robot and the measurement results from thevisual sensor simulator 150) output from the controller simulator 160.More specifically, the reproduction module 170 uses thethree-dimensional design data 172, which is a definition file, tovisualize the system virtually created in the three-dimensional virtualspace, and also uses the trace data provided from the controllersimulator 160 to reproduce the chronological changes of the workpieceand the robot in the system.

The reproduction module 170 includes a workpiece display setting module171. The workpiece display setting module 171 refers to the workpiecedisplay setting parameter 174 to determine the display mode of theworkpiece to be reproduced. More specifically, the workpiece displaysetting module 171 sets and changes the color, shape, size, andorientation of the workpiece to appear in the three-dimensional virtualspace in accordance with the workpiece display setting parameter 174.The information added to the workpiece to be reproduced will bedescribed in detail later.

In this manner, the reproduction module 170 represents the chronologicalchanges of the simulation results in the form of animation or a movingimage on the display unit 108 of the simulator 100 (FIG. 3).

In the functional structure shown in FIG. 4, the controller simulator160 outputs the time-series data for its output control instruction tothe robot and also the trace data including the measurement results fromthe visual sensor simulator 150. However, the functional structure isnot limited to this example. The reproduction module 170 may combine thetime-series data with the trace data to reproduce the system behavior.

FIG. 5 is a schematic diagram showing the functional structure of thesimulator 100 according to a modification of the present embodiment. Thesimulator 100 shown in FIG. 5 includes a measurement result storage unit130 as a software function, in addition to the functional modules shownin FIG. 4. The measurement result storage unit 130 sequentially storesthe measurement results from the image measurement performed by thevisual sensor simulator 150 and the corresponding encoder values. Inaddition to the measurement results and the encoder values, themeasurement result storage unit 130 may also store the input images thathave undergone the image measurement.

The reproduction module 170 reproduces, in the three-dimensional virtualspace, the chronological changes of the workpiece and the robot in thesystem by displaying the workpiece associated with the results from eachimage measurement stored in the measurement result storage unit 130based on the corresponding encoder value.

With the functional structure shown in FIG. 5, for example, an inputimage that has undergone image measurement may also be displayedtogether with the reproduced behavior of the workpiece.

Although FIGS. 4 and 5 show the example in which the reproduction module170 reproduces the behavior of the created system using the trace dataoutput from the controller simulator 160, the simulator 100 may notinclude the reproduction module 170. For example, the trace data fromthe controller simulator 160 may be output to an external device or anexternal application, and the external device or the externalapplication may reproduce the behavior of the system. In someembodiments, the reproduction module 170 may simply generate and storemoving image data for reproducing the behavior of the system in anystorage medium, which may then be reproduced by another application.

The encoder emulator 190 generates information indicating the positionor displacement of the carrier in a manner associated with the movementof the carrier. In one example, the encoder emulator 190 may output theencoder value indicating a displacement from a reference position, ormay generate pulses proportional to a movement of the carrier (conveyor)per unit time. In this case, the encoder value indicates the position ofthe conveyor, and the number of pulses per unit time indicates the speedof the conveyor.

D. Procedure

The procedure of simulation performed by the simulator 100 according tothe present embodiment will now be described.

FIG. 6 is a flowchart showing the procedure of simulation performed bythe simulator 100 according to the present embodiment. With reference toFIG. 6, the simulator 100 first receives the settings of the systemmodel (step S2). The settings of the system model include thearrangement of the devices included in the system, and the moving speedof the conveyor, which is a carrier. Based on these system modelsettings, the simulator 100 (model creating module 182) virtuallycreates a system to be simulated (system model) in a three-dimensionalvirtual space.

The simulator 100 (user interface module 180) receives an imaging areafor a visual sensor defined in the system model (step S4). Based on therelative positional relationship between the created system and thedefined imaging area, the simulator calculates a calibration parameter,which is a conversion parameter for transforming the measurement resultsinto an input value for a control operation.

The simulator 100 (user interface module 180) then receives a controlprogram for controlling the system model (step S6). The control programis associated with the system, and is to be executed by the controllersimulator 160.

The simulator 100 (user interface module 180) receives the settings forimage measurement to be performed by the visual sensor simulator 150(step S8). The settings include designation of the processing details ofthe image measurement and reference information (e.g., a model image,and a feature quantity calculated from the model image) associated withthe designated processing details.

This procedure completes the settings for the simulation.

When instructed to start the simulation, the simulator 100 (encoderemulator 190) updates an encoder value indicating the position ormovement of a virtual conveyor at specified time intervals (step S10).The simulator 100 (controller simulator 160) determines whether acondition for generating a trigger signal is satisfied (step S12). Whenthe condition is satisfied (Yes in step S12), the simulator 100virtually generates a trigger signal (step S14). When the condition isnot satisfied (No in step S12), the processing in step S14 is skipped.

In response to the generated trigger signal, the simulator 100 (visualsensor simulator 150) retrieves the corresponding image data from thepreliminarily obtained image data group (step S100), and performs theimage measurement (step S102). After the image measurement, thesimulator 100 (visual sensor simulator 150) outputs the measurementresults (step S104). The processing in steps S100 to S104 is performedindependently of the processing performed in the controller simulator160.

Subsequently, the simulator 100 (controller simulator 160) determineswhether the measurement results from the image measurement have beenupdated (step S16). More specifically, the simulator 100 determineswhether new measurement results have been received from the visualsensor simulator 150. When the measurement results have been updated(Yes in step S16), the simulator 100 (controller simulator 160) performsa control operation based on the updated measurement results (step S18).When the measurement results have not been updated (No in step S16), theprocessing in step S18 is skipped.

The simulator 100 (controller simulator 160) stores values calculatedthrough the control operation in a manner associated with thecorresponding encoder values, which are chronological information (stepS20).

The simulator 100 determines whether a preset simulation period hasended (step S22). When the simulation period has not ended (No in stepS22), the processing in step S10 and subsequent steps is repeated.

When the preset simulation period has ended (Yes in step S22), thesimulator 100 reproduces the behavior of the system model using thetrace data sequentially stored in step S20 (step S24).

When the setting of the workpiece display mode used to reproduce thebehavior of the system model is changed in accordance with a useroperation (Yes in step S26), the simulator 100 reproduces the behaviorof the system model in the newly set workpiece display mode (step S24).

The simulator 100 can also change the time intervals and the updatefrequency of the behavior of the reproduced system model as appropriatein accordance with a user operation.

With the procedure described above, the processing capability (e.g., atact time) and the accuracy of processing in the system model can beevaluated preliminarily.

E. Image Measurement

The image measurement to be performed in the simulator according to thepresent embodiment includes searching an input image for a partcorresponding to one or more pieces of predetermined referenceinformation. An example of such image measurement will now be described.The image measurement is not limited to this example, and may be any ofvarious other examples of image measurement.

FIGS. 7A and 7B are schematic diagrams describing examples of the imagemeasurement performed by the simulator 100 according to the presentembodiment. FIG. 7A shows the processing suitable for determining thetype of a workpiece, whereas FIG. 7B shows the processing suitable fordetermining the quality of a workpiece.

Referring now to FIG. 7A, for example, a plurality of model imagescaptured by imaging workpieces to be detected are registered aspredetermined reference information. The degrees of correspondence(typically, correlation values) between the preliminarily registeredmodel images and the input images sequentially captured by imagingworkpieces on a transporting path are calculated sequentially. Among thecalculated correlation values between the input images and the modelimages, the correlation value that is the highest and exceeds apredetermined threshold is determined. The type of the workpiece is thendetermined based on the model image corresponding to the determinedcorrelation value.

Thus, the workpiece type includes information indicating the mostprobable degree of correspondence with the workpiece among thepredetermined model images (reference information). The referenceinformation may be input images captured by imaging sample workpieces orfeature images representing feature quantities (e.g., edge values)extracted from the input images.

This image measurement can be used to determine the type of eachworkpiece in a system for processing a mixture of different types ofworkpieces transported on a belt conveyor.

Referring now to FIG. 7B, for example, a model image captured by imaginga non-defective workpiece is registered as predetermined referenceinformation. The degrees of correspondence (typically, correlationvalues) between the preliminarily registered model images and the inputimages sequentially captured by imaging workpieces on a transportingpath are calculated. When each calculated correlation value with themodel image exceeds a predetermined threshold, the target workpiece isdetermined to be non-defective. In any other cases, the target workpieceis determined to be defective.

Thus, the workpiece type includes information indicating whether thedegree of correspondence between the workpiece and the predeterminedmodel image (reference information) satisfies a predetermined condition.The reference information may be an input image captured by imaging asample workpiece or a feature image representing a feature quantity(e.g., an edge value) extracted from the input image.

This image measurement can be used to determine whether each workpieceis non-defective or defective in a system involving a plurality ofworkpieces transported on a belt conveyor.

FIG. 8 is a schematic diagram describing the measurement results of theimage measurement performed by the simulator 100 according to thepresent embodiment. FIG. 8 shows an example of pattern matching of aninput image with a predetermined model image.

In FIG. 8, an object included in an input image is determined tocorrespond to its model image. The measurement results (x, y, θ)(type/quality/correlation value) are output. More specifically, themeasurement results of the image measurement typically include (1) thecoordinates (x, y) indicating the center of a part (object) detected inan input image, (2) the rotation angle θ of the detected part of theinput image with respect to the model image, and (3) the type of themodel image matching the part in the input image. The rotation angle θis an angle by which an input image part corresponding to any modelimage (reference information) is rotated.

As described above, the type indicating the model image matching aworkpiece may be replaced with information indicating whether the degreeof correspondence with a particular model image satisfies apredetermined condition (or exceeds a threshold) (defective ornon-defective), or a value indicating the degree of correspondence witha particular model image (correlation value).

The controller simulator 160 (refer to FIGS. 4 and 5) for simulating theprocessing in the controller 200 transforms the measurement resultsshown in FIG. 8 from the visual sensor simulator 150 (refer to FIGS. 4and 5) for simulating the processing performed by the visual sensor 220into the coordinates in a coordinate system of the system model, andthen generates a control instruction. The controller simulator 160 alsogenerates a control instruction for any processing based on the detectedrotation angle of each workpiece (e.g., the processing for orientingeach workpiece in the same direction).

F. Workpiece Display Position in Reproducing Simulation Results

The processing for calculating the position of a workpiece when thebehavior of a system model is reproduced by the simulator 100 accordingto the present embodiment will now be described. More specifically, thesimulator 100 (reproduction module 170) sequentially updates the displayposition of a workpiece in a three-dimensional virtual space based oninformation indicating the position or displacement of the conveyortransporting the workpiece.

As described with reference to FIG. 8, the measurement results of imagemeasurement include the coordinates (x, y) indicating the center of apart (object) detected in the input image. The coordinates (x, y), whichare values in a local coordinate system used for image measurement, areto be transformed into the coordinates in a three-dimensional virtualspace.

More specifically, the simulator 100 can use transform coefficients A toF for transforming the coordinates (x, y) of an input image defined inthe camera coordinate system used in image measurement into thecoordinates (X, Y) defined in a world coordinate system defining thethree-dimensional virtual space. The simulator 100 can thus calculatethe initial display position at the time of input into the controllersimulator 160 based on the workpiece coordinates (x, y) detected in thevisual sensor simulator 150 in the manner described below.Workpiece initial display position X0=A×x+B×y+CWorkpiece initial display position X0=D×x+E×y+F

A movement Xd of the conveyor in X-direction and a movement Yd of theconveyor in Y-direction per pulse of an encoder value can be used tocalculate the workpiece display position corresponding to a displacementEt indicated by the encoder value as written in the formulas below.Workpiece display position (X)=Xd×Et+X0Workpiece display position (Y)=Yd×Et+Y0

When the absolute value of an encoder value is used, a deviation fromthe encoder value for each workpiece displayed initially may beincorporated in these formulas.

The simulator 100 sequentially updates the display position of eachworkpiece in accordance with these formulas.

G. Visualizing Simulation Results

The processing for visualizing the behavior of the system modelperformed by the simulator 100 according to the present embodiment willnow be described. In the present embodiment, the behavior of a workpiecedetected in image measurement is reproduced in the system model togetherwith information about the type and the orientation of the workpiece.

FIG. 9 is a diagram showing an example user interface screen forreproducing simulation results provided from the simulator 100 accordingto the present embodiment. In the user interface screen shown in FIG. 9,an object included in a three-dimensional virtual space can be renderedin any direction. More specifically, a user can freely change a point ofview rendered in the user interface screen.

In the system model shown in FIG. 9, the conveyor 230 transporting aworkpiece to be picked up and the conveyor 240 transporting a workpieceplaced on it are arranged in parallel. The conveyors 230 and 240 areassociated with two robots 311 and 313. In this system model, aworkpiece 232 is transported by the conveyor 230 from left to right inthe drawing. When the workpiece 232 reaches the predetermined trackingarea 231 or 233, the robot 311 or 313 picks up the incoming workpiece232 and places the workpiece on the conveyor 240. The robots 311 and 313each place the workpiece 232 on the corresponding tracking area 235 or237 defined on the conveyor 240. Each workpiece 232 placed in a randomorientation on the conveyor 230 is aligned in a predetermined directionwhen placed on the conveyor 240.

In one example application, the conveyor 230 may transport at least twotypes of workpieces 232. The robot 311 is controlled to pick up andplace one specific type of workpiece, whereas the robot 313 iscontrolled to pick up and place the other type of workpiece. Thedifferent types of workpieces may have different shapes. In this case, arobot having a special tool dedicated to a particular type of workpiecemay be used for that type of workpiece.

Each workpiece 232 has information about the type and the orientation ofthe workpiece.

In the user interface screen shown in FIG. 9, the type of a workpiece232 can be identified by the appearance color of the workpiece 232. Morespecifically, the appearance color of a workpiece 232 to be picked andplaced by the robot 311 is different from that of a workpiece 232 to bepicked and placed by the robot 313. As described above, the workpiecetype may include the quality of the workpiece, in addition to theinformation indicating the model image matching the target workpiece.For example, a workpiece determined to be non-defective and a workpiecedetermined to be defective may have different appearance colors.

Additionally, the information about the orientation of the workpiece 232includes two coordinate axes 320 indicating the rotation angle includedin the measurement results. The coordinate axes 320 indicate therotation angle of a workpiece 232 that has undergone image measurement,with respect to the origin of coordinates in the imaging area 221defined in a part of the conveyor 230. More specifically, the rotationangle θ in FIG. 8 described above is transformed into values in acoordinate system defining the system model. These values correspond tolocal coordinates.

The use of the coordinate axes 320 representing the orientation of aworkpiece 232 enables a user to determine whether the control programfor controlling the robots is running correctly. Further, the coordinateaxes 320 can indicate the orientation of a workpiece 232, which isexpressed by a rotation by an angle of up to 360 degrees. This enablesdetermination as to, for example, whether a cubic workpiece 232 has thesame orientation as the model image (rotation angle=0 degrees) or hasthe reverse orientation (rotation angle=180 degrees).

The user interface screen shown in FIG. 9 may also display movementareas 312 and 314, in which the robots 311 and 313 can move, based onthe design information for the two robots 311 and 313. The movementareas 312 and 314 can be used to preliminarily examine an optimuminterval between the adjacent robots.

Additionally, the tracking areas 231 and 233 defined on the conveyor 230and the tracking areas 235 and 237 defined on the conveyor 240 are alsodisplayed. This further enables visual checking of the areas in whichthe robots 311 and 313 are movable (or the range within which aworkpiece can be picked up and placed).

To reproduce the behavior of each workpiece on the user interface screenshown in FIG. 9, a workpiece is displayed at the timing when measured bythe visual sensor simulator 150 (at the timing identified using anencoder value indicating the movement of the conveyor 230 during thesimulation). The display position of the workpiece is subsequentlyupdated as the encoder value is updated (incremented). For a workpiecepicked up and placed on the conveyor 240 by the robot, its displayposition is sequentially updated based on the encoder value indicatingthe movement of the conveyor 240. When the value indicating the displayposition of the workpiece shows that the workpiece has moved out of therange of the conveyor 240, or in other words when the workpiece hasreached the end of the conveyor 240, the workpiece disappears.

Although FIG. 9 shows the user interface screen in one example forreproducing the simulation according to the present embodiment, theinformation about the type and the orientation of a workpiece may appearin any display mode and may be set with any method as described below.

g1: Display Mode for Workpiece Type

The simulator 100 (reproduction module 170) according to the presentembodiment displays each type of workpiece with at least one of adifferent color, a different shape, or a different size. In the displaymodes shown in FIG. 9, different workpiece types are distinguished bydifferent colors. In some embodiments, objects as workpieces may havedifferent shapes depending on their types, instead of or in addition tohaving different colors.

FIGS. 10A and 10B are schematic diagrams describing an example of usersetting for the display mode of simulation results provided from thesimulator 100 according to the present embodiment. For example, FIG. 10Ashows a setting screen 400, which allows the user to set any workpiecedisplay mode used for reproducing the simulation results. Thus, thesimulator 100 may have the user interface module 180 (input unit) forreceiving the setting of at least one of the color, shape, and size usedfor displaying each type of workpiece.

The setting screen 400 includes a pull-down list 402, which is used toselect the display mode of each workpiece type. When an entry in thepull-down list 402 is selected, the corresponding pull-down menu 404appears.

The pull-down menu 404 includes a color choice area 406 for choosing thecolor of target workpieces and a shape choice area 408 for choosing theshape of the target workpieces. The user chooses one of the colorslisted in the color choice area 406 and one of the shapes listed in theshape choice area 408. In this manner, the user can choose the color andthe shape for each type of workpiece, for which the behavior is to bereproduced.

After the color and the shape are designated in this manner, thesimulation results are reproduced in the display mode shown in FIG. 10B.

In some embodiments, the user may freely set the sizes of workpieces.FIG. 11 is a schematic diagram describing another example of usersetting for the display mode of simulation results provided from thesimulator 100 according to the present embodiment. For example, FIG. 11shows a setting screen 410, on which the user selects an entry in apull-down list 412 to display the corresponding pull-down menu 414.

The pull-down menu 414 includes a color choice area 416 for choosing thecolor of target workpieces and a size entry area 418 for setting thedimensions (length, width, and height) of target workpieces. The userchooses one of the colors listed in the color choice area 416, and setsthe dimensions of workpieces to be displayed in the size entry area 418.

In the size entry area 418, the actual dimensions of workpieces may beentered directly. In this case, the system model virtually created in athree-dimensional space is also associated with its actual dimensions,and thus the workpieces are represented with the correspondingdimensions in accordance with a conversion parameter calculated from thedimensions of the system model.

The dimensions of workpieces may be freely set in this manner. Thus, theuser can check the simulation results nearer the reality in accordancewith the application to be simulated.

Additionally, the user may freely set an image used for reproducingworkpieces. FIG. 12 is a schematic diagram describing another example ofuser setting for the display mode of simulation results provided fromthe simulator 100 according to the present embodiment. FIG. 12 shows asetting screen 420, on which the user selects a setting button 422 todisplay the corresponding image selection menu 424.

The image selection menu 424 lists images used for displaying targetworkpieces, together with the corresponding file names 426. The userchooses an image file from the listed file names 426. When the userselects an OK button 428, the selected image file is activated. As aresult, the workpieces of different types are reproduced using thecorresponding images included in the designated image file.

An image used for displaying workpieces may be freely set in thismanner. Thus, the user can check the simulation results nearer thereality in accordance with the application to be simulated.

The simulation results may be reproduced in a mode to allow the user toreadily determine the type indicated by the type information assigned toeach workpiece.

FIGS. 13A and 13B are diagrams each showing an example user interfacescreen for supporting determination of each workpiece type based onsimulation results provided from the simulator 100 according to thepresent embodiment.

The user interface screen shown in FIG. 13A shows workpieces 232 and alegend 340 specifying the workpiece types corresponding to the colors ofthe workpieces 232. The user can refer to the correspondence indicatedby the legend 340 and readily determine the type of each workpiece 232.

The user interface screen shown in FIG. 13B shows a label 342 associatedwith each workpiece 232, such as Type 1 and Type 2. The user can referto each label 342 and readily determine the type of each workpiece 232.

The workpiece displays are not limited to the examples shown in FIGS.13A and 13B. Any support display may be used to allow the user toreadily determine the type of each workpiece. The display examplesdescribed above may also be combined as appropriate.

g2: Display Mode of Workpiece Orientation

The simulator 100 (reproduction module 170) according to the presentembodiment generates information about the orientation of a workpiecebased on the rotation angle included in the measurement results.Although FIG. 9 is a diagram showing the display mode showing theorientation of a workpiece using the two coordinate axes 320 as objects,any other display mode may be used.

FIGS. 14A and 14B are diagrams each showing an example display ofworkpiece orientations based on simulation results provided from thesimulator 100 according to the present embodiment.

The display example shown in FIG. 14A shows marks 360 on workpieces 232.Each mark 360 represents a reference direction. The reference directionmay be, for example, the direction with the rotation angle of 0 degreeswith respect to the corresponding model image. Referring to FIG. 8, thereference direction is determined to be the same direction as set for apredetermined model image. A mark 360 is assigned to a part of aworkpiece 232 oriented in the reference direction. The mark 360 thusallows the user to readily determine the orientation of each workpiece232.

The display example shown in FIG. 14B includes the coordinate axes 320as well as labels 362 indicating the rotation angles of workpieces 232.Each label 362 indicates the angle of a direction in which thecorresponding workpiece 232 is oriented. Although both the coordinateaxes 320 and the labels 362 are shown in the display example in FIG.14B, only the labels 362 may be displayed.

In this manner, objects indicating the orientations of workpieces(coordinate axes 320, marks 360, or labels 362) may be displayedtogether with the workpieces.

In some embodiments, the shape of a workpiece used for display may beassociated with the orientation of the workpiece. FIGS. 15A and 15B arediagrams each showing another example display of workpiece orientationsbased on simulation results provided from the simulator 100 according tothe present embodiment.

In the display example shown in FIG. 15A, an area of a workpiece 232corresponding to a reference direction has a display mode different fromthat of the other area. More specifically, an area 364 of each workpiece232 corresponding to a reference direction (one side of each prism inthe display example shown in FIG. 15A) has a color different from thecolor of the other area. This display including the area 364 allows theuser to readily determine the direction in which each workpiece 232 isoriented.

In FIG. 15A, the display mode of one side of each prism differs fromthat of the other area. However, the display is not limited to thisexample. For example, a particular surface of a prism may have a displaymode different from that of the other area.

In the display example shown in FIG. 15B, an area of a workpiece 232corresponding to a reference direction has a shape different from theshape of the other area. More specifically, an area 366 of eachworkpiece 232 corresponding to a reference direction (an edge of eachprism in the display example shown in FIG. 15B) is chamfered. The shapeof the area 366 allows the user to readily determine the direction inwhich each workpiece 232 is oriented. Additionally, the chamfered edgeof the area 366 may have a color different from the color of the otherarea.

In this manner, a feature (area 364 or 366) may be added to theappearance of a reproduced workpiece in a manner associated with theorientation of the workpiece.

The display mode is not limited to the examples shown in FIGS. 14 and15, and any display mode may be used to allow the user to readilydetermine the orientation of a workpiece. The display examples describedabove may also be combined as appropriate.

g3: Additional Objects

Although the display modes of the workpieces detected in input imagesthrough image measurement have been described, not only workpieces butalso additional objects may be displayed to allow more accuratesimulation near an actual application. In the example described below,the simulation results with additional objects are reproduced.

For example, the pick-and-place operation has applications includingplacement of one or more workpieces into a case, such as packing. A boxin which the workpieces are packed may also be displayed in athree-dimensional virtual space. The box is also transported on aconveyor, and thus this behavior can be visualized to performpreliminarily evaluation to determine whether the conveyor and therobots operate as designed.

FIG. 16 is a diagram showing another example user interface screen forreproducing simulation results provided from the simulator 100 accordingto the present embodiment. With reference to FIG. 16, multiple referencelines 260 are displayed on the conveyor 240 at intervals preset by theuser. The intervals between the displayed reference lines 260 are, forexample, preset based on required specifications (including a tacttime). When simulation results are reproduced, the reference lines 260appear on the conveyor 240 at the preset intervals.

For example, the reference lines 260 may serve as references to defineareas in which workpieces are to be placed by the pick-and-placeoperation. The user can evaluate whether the workpieces 232 on theconveyor 240 are accurately placed at positions indicated by thereference lines 260.

Objects 250 indicating boxes used in an actual application may also bedisplayed instead of or in addition to the reference lines 260. Thedisplay positions of and the display intervals between the objects 250may also be preset by the user in the same manner as for the referencelines 260. Further, the shape of the objects 250 may also bepreliminarily and freely set by the user. In the same manner as for theappearance of workpieces set as described with reference to FIGS. 10 to12, the shape, color, and size of the objects 250 may be set freely. Theobjects 250 may be transparent or semitransparent to allow the user toreadily check the positional relationship between the objects 250 andthe workpieces 232.

As shown in FIG. 16, the additional objects 250 and/or the referencelines 260 are virtually displayed in addition to the workpieces 232 toallow visual evaluation to determine whether the workpieces areprocessed as designed.

H. Modification

Although the above embodiment illustrates a typical example in whichtargets are transported on a transporting path, the embodiment isapplicable to another system as described below.

For example, when workpieces undergo multiple processes, multiple robotsinstalled in a line may cooperate with each another to implement anintended operation, or humans and robots may cooperate with each otherto implement an intended operation. To transfer a workpiece betweenprocesses in such an arrangement, a common work area is provided betweena robot or a human in an upstream process and a robot in a downstreamprocess. The common work area has the function of simply buffering aworkpiece, instead of using a carrier such as a conveyor. A system withan imaging area for a visual sensor defined in such a common work areacan be virtually created in a three-dimensional virtual space with theprocedure described in the above embodiment. This allows visualevaluation to determine whether workpieces are processed as designed inthe system.

I. Advantages

The simulator 100 according to the present embodiment allows evaluationof the behavior of the overall system for processing workpieces bychecking both the positions and the orientations of the workpieces thatare processed by the processing device such as robots in accordance witha control instruction. This structure allows the validity of the systemunder examination to be determined readily.

The embodiments disclosed herein should be considered to be in allrespects illustrative and not restrictive. The scope of the presentinvention is determined not by the description given above but by theclaims, and is construed as including any modification that comes withinthe meaning and range of equivalency of the claims.

REFERENCE SIGNS LIST

1, 2 conveyor tracking system

100 simulator

102 processor

104 main memory

106 input unit

108 display unit

110 network interface

112 optical drive

114 optical disc

116 communication interface

118 internal bus

120 hard disk drive

122 OS

124 simulation program

130 measurement result storage unit

140 image data group

150 visual sensor simulator

152 setting parameter

160 controller simulator

162 control program

170 reproduction module

171 workpiece display setting module

172 three-dimensional design data

174 workpiece display setting parameter

180 user interface module

182 model creating module

190 encoder emulator

200 controller

202 network

210, 311, 313 robot

220 visual sensor

221 imaging area

222 imaging unit

230, 240 conveyor

231, 233, 235 tracking area

232 workpiece

234, 244 drive roller

236, 246 encoder

250 object

260 reference line

400, 410, 420 setting screen

402, 412 pull-down list

404, 414 pull-down menu

406, 416 color choice area

408 shape choice area

418 size entry area

422 setting button

424 image selection menu

The invention claimed is:
 1. A simulator for estimating a behavior of asystem comprising a processing device for processing a target, thesimulator comprising a processor configured with a program to performoperations comprising: operation as a creating unit configured tovirtually create the system in a three-dimensional virtual space;operation as a measurement unit configured to perform image measurementof an input image comprising at least a part of the target, the inputimage associated with a first area predefined at a predeterminedposition in the three-dimensional virtual space, the image measurementcomprising searching the input image for a part corresponding to one ormore pieces of predetermined reference information; operation as anexecution unit configured to execute a control operation for generatinga control instruction directed to the processing device based on ameasurement result obtained by the measurement unit, the measurementresult comprising a type of the target and a quality of the target; andoperation as a reproduction unit configured to reproduce, in thevirtually created system, a behavior of the target based on time-seriesdata for the control instruction, the measurement result, and anorientation of the detected target.
 2. The simulator according to claim1, wherein the type of the target comprises information indicating apiece of reference information having a most probable degree ofcorrespondence with the target.
 3. The simulator according to claim 1,wherein the type of the target comprises information indicating whethera degree of correspondence between the target and the pieces ofpredetermined reference information satisfies a predetermined condition.4. The simulator according to claim 1, wherein the processor isconfigured with the program such that operation as the reproduction unitcomprises operation as the reproduction unit that displays the targetusing at least one of: a color; a shape; or a size that differ dependingon the type of the target.
 5. The simulator according to claim 4,wherein the processor is configured with the program to performoperations further comprising operation as an input unit configured toreceive a setting of at least one of the color, the shape, or the sizethat differ depending on the type of the target.
 6. The simulatoraccording to claim 1, wherein the processor is configured with theprogram such that: operation as the measurement unit comprises operationas the measurement unit that outputs the measurement result comprising arotation angle of the part corresponding to one or more pieces ofpredetermined reference information included in the input image; andoperation as the reproduction unit comprises operation as thereproduction unit that generates information about the orientation ofthe target based on the rotation angle output as the measurement result.7. The simulator according to claim 1, wherein the processor isconfigured with the program such that operation as the reproduction unitcomprises operation as the reproduction unit that displays, in additionto the target, an object indicating the orientation of the target. 8.The simulator according to claim 1, wherein the processor is configuredwith the program such that operation as the reproduction unit comprisesoperation as the reproduction unit that displays a feature added to anappearance of the reproduced target with the orientation of the target.9. The simulator according to claim 1, wherein the system furthercomprises a carrier configured to transport the target, and thepredetermined position at which the first area is predefined comprises atransporting path of the carrier.
 10. The simulator according to claim9, wherein the processor is configured with the program such thatoperation as the reproduction unit comprises operation as thereproduction unit that sequentially updates a display position of thetarget in the three-dimensional virtual space based on informationindicating a position or a displacement of the carrier that transportsthe target.
 11. A simulation method implemented by a computer forestimating a behavior of a system comprising a processing device forprocessing a target, the method comprising: virtually creating thesystem in a three-dimensional virtual space; performing imagemeasurement of an input image comprising at least a part of the target,the input image associated with a first area predefined at apredetermined position in the three-dimensional virtual space, the imagemeasurement comprising searching the input image for a partcorresponding to one or more pieces of predetermined referenceinformation; executing a control operation for generating a controlinstruction directed to the processing device based on a measurementresult from the image measurement, the measurement result comprising atype of the target and a quality of the target; and reproducing, in thevirtually created system, a behavior of the target based on time-seriesdata for the control instruction, the measurement result, and anorientation of the detected target.
 12. A non-transitorycomputer-readable medium storing a simulation program for estimating abehavior of a system comprising a processing device for processing atarget, the simulation program causing a computer to implement:virtually creating the system in a three-dimensional virtual space;performing image measurement of an input image comprising at least apart of the target, the input image associated with a first areapredefined at a predetermined position in the three-dimensional virtualspace, the image measurement comprising searching the input image for apart corresponding to one or more pieces of predetermined referenceinformation; executing a control operation for generating a controlinstruction directed to the processing device based on a measurementresult from the image measurement, the measurement result comprising atype of the target and a quality of the target; and reproducing, in thevirtually created system, a behavior of the target based on time-seriesdata for the control instruction, the measurement result, and anorientation of the detected target.
 13. The simulation method accordingto claim 11, wherein the type of the target comprises informationindicating a piece of reference information having a most probabledegree of correspondence with the target.
 14. The simulation methodaccording to claim 11, wherein the type of the target comprisesinformation indicating whether a degree of correspondence between thetarget and the pieces of predetermined reference information satisfies apredetermined condition.
 15. The simulation method according to claim11, wherein reproducing, in the virtually created system, a behavior ofthe target further comprises displaying the target using at least oneof: a color; a shape; or a size that differ depending on the type of thetarget.
 16. The simulation method according to claim 15, furthercomprising receiving a setting of at least one of the color, the shape,or the size that differ depending on the type of the target.
 17. Thesimulation method according to claim 11, further comprising: outputtingthe measurement result comprising a rotation angle of the partcorresponding to one or more pieces of predetermined referenceinformation included in the input image, wherein reproducing, in thevirtually created system, a behavior of the target further comprisesgenerating information about the orientation of the target based on therotation angle output as the measurement result.
 18. The simulationmethod according to claim 11, wherein reproducing, in the virtuallycreated system, a behavior of the target further comprises displaying,in addition to the target, an object indicating the orientation of thetarget.
 19. The simulation method according to claim 11, reproducing, inthe virtually created system, a behavior of the target further comprisesdisplaying a feature added to an appearance of the reproduced targetassociated with the orientation of the target.
 20. The simulation methodaccording to claim 11, wherein the system further comprises a carrierconfigured to transport the target, and the predetermined position atwhich the first area is predefined comprises a transporting path of thecarrier.